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1.0 INTRODUCTION 

The complexity of modem military systems, as well as the cost and difficulty associated with experimentally 
verifying system and subsystem design makes the use of high-fidelity based simulation a future alternative for 
design and development. The predictive ability of such simulations such as computational fluid dynamics 
(CFD) and computational structural mechanics (CSM) have matured significantly. However, for numerical 
simulations to be used with confidence in design and development, quantitative measures of uncertainty must 
be available. 

The AVT 147 Symposium has been established to compile state-of-the art methods of assessing 
computational uncertainty, to identify future research and development needs associated with these methods, 
and to present examples of how these needs are being addressed and how the methods are being applied. 

Papers were solicited that address uncertainty estimation associated with high fidelity, physics-based 
simulations. The solicitation included papers that identify sources of error and uncertainty in numerical 
simulation from either the industry perspective or from the disciplinary or cross-disciplinary research 
perspective. Examples of the industry perspective were to include how computational uncertainty methods are 
used to reduce system risk in various stages of design or development. 


2.0 CHARACTERIZATION OF CONTRIBUTIONS 

The authors of this evaluation decided to divide the report into sections depending on the primary aspect of 
method quality being addressed. The aspects were divided into method qualification, i.e. code verification and 
model validation, and prediction qualification, i.e. solution verification, uncertainty analysis, uncertainty 
propagation, and sensitivity analysis. The definitions are given below: 


1 Head, Institute of Fluid Mechanics, Technical University Braunschweig. 

2 Aerospace Engineer, NASA Langley Research Center. 
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• Method Qualification 

o Code verification - The process of gathering evidence that the mathematical model as 
implemented and its solution are correct. 

o Model validation - The quantitative determination that the model is an accurate representation of 
the real world for its intended use. 

* Prediction Qualification 

o Solution verification - The estimation of residual discretization error. 

o Uncertainty analysis - The identification and characterization of uncertainty sources. 

o Uncertainty propagation - The propagation of the uncertainties into the system response 
quantities. 

o Sensitivity analysis - The determination of the most important uncertainty contributors. 

A table showing the breakdown of the papers according to this characterization is given at the end of the 
evaluation report. 


3.0 REVIEW OF PAPERS 

The AVT 147 Symposium was organised in 3 plenary sessions and 10 thematically focussed sessions held in 
two parallel streams. The following reviews cover all papers presented during these sessions. 

3.1 Plenary Sessions 

3.1.1 Plenary 1 - Airframe Perspective 

Plenary Paper 1- P. Raj: Computational Uncertainties - Achilles ’ Heel of Simulation Based Aircraft Design 

Starting from a historical review of aircraft development Dr. Raj describes the motivation of airframe industry 
to achieve technological superiority at affordable cost, particularly since the end of the cold war. Aircraft 
design has a strong impact on overall life cycle cost, but this is often not appreciated by the design teams. This 
situation motivates new paradigms of integrated multidisciplinary design using proper computational 
simulations in order to accomplish timely reduction of uncertainties. 

Integrated computer aided methods enable simulation based design. Key areas in Computer Aided Design 
(CAD) are manufacture and maintenance cost representations and parametric design capabilities. Computer 
Aided Manufacture (CAM) methods aim for simulation of manufacturing processes and production systems. 
Finally, Computer Aided Engineering (CAE) methods allow configuration performance analysis including 
multidisciplinary interactions. They are the key to design optimization. Based on co mm on views on 
effectiveness, measures of quality and acceptance factors of CAE methods are presented. The role of CFD 
methods for new products is reviewed in more detail regarding performance assessment, structure design, 
flight control and optimization. Highlights of present capabilities are reduction of drag hot spots, vortical flow 
calculation of tail buffet and propulsion integration of transport aircraft. 

The draw back of today’s CFD simulation is that a capability to obtain computational uncertainties is still not 
available. While high-fidelity physics are continuously being integrated, there is a need to compute 
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uncertainty bands as well. New approaches are needed for validation in order to achieve true predictive 
capabilities, in particular for unsteady flows. Overcoming these problems is viewed as the decisive factor to 
the success of simulation based design. 

3.1.2 Plenary 2 - Powerplant Perspective 

Plenary Paper 2- S. Shahpar: Toward Robust CFD based Design Optimisation of Virtual Engine 

The environmental challenges force significant reductions in noise, fuel consumptions and NOX emissions 
which often lead to conflicting design requirements. The power of simulation in turbomachinery design is 
apparent from the decreasing number of tests required in previous and current Rolls Royce products. Blind 
test cases, however, point to the need of reducing computational uncertainties before the virtual engine is 
accomplished. An important technology driver is the high cost to recover design failures along the 
development chain. 

Source of errors in CFD simulation for turbomachinery are presented in more detail by describing the 
associated flow phenomena. Requirements for proper design and optimization methods are also highlighted 
where industrial emphasis is on parametric design using automated processes. Minimizing losses in 
compressors with numerous design parameters calls for robustness of optimization approaches. While the 
CFD-suite F1YDRA offers many modelling options there remain significant sources of uncertainty in the 
numerical solutions, particularly related to grids. 

A study case on turbine stage design presents the challenge of optimizing losses along with simulating 
increasing fidelity of the geometry. These designs ask for robust design methods and smart utilization of 
computational resources. Additional issues are the prediction of flutter, prediction of hot structures, and 
analysis of manufacture accuracy which emphasize the need for multidisciplinary simulation based design. 

3.1.3 Plenary 3 - Verification and Validation 

Plenaiy Paper 4- W. Oberkampf S. Ferson: Model Validation under both Aleatoiy and Epidemistic 
Uncertainty 

The three aspects of validation concern (1) assessment of model accuracy by comparison with experimental 
data, (2) processes of interpolation and extrapolation to intended use, and finally, (3) decision of model 
adequacy. Existing approaches involve quantitative comparisons, hypothesis testing based on probability 
values, and Bayesian validation that aims for updating probability density functions of uncertain data. The 
need is emphasized for new approaches that are based on Cumulative Distribution Functions (CDF) between 
simulation and experiment and that can estimate the mismatch of the system response quantity. 

The validation metric proposed here is defined as the area between the CDF and measured data points which 
can be few. The area metric can be interpreted as the minimum over all discrepancies between the measured 
and experimental distributions. Problems arise when data replicates are combined. This is solved by 
transforming individual system response quantities to probability space, combining and back-transforming to 
physical space. 

Aleatory uncertainty is defined as variation associated with parameters, physical systems or environment 
(stochastic, irreducible). Epistemic uncertainty arises from imperfect knowledge or ignorance that could be 
reduced in principle (poor understanding of physics phenomena or failure modes). Large epistemic 
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uncertainties can result in zero validation metric (no evidence for disagreement between simulation and 
experiment). In that sense the validation metric describes evidence of mismatch. Oberkampf describes a 
generalized validation metric that takes into account aleatory and epistemic uncertainties and presents the 
results when these uncertainties vary. This approach forms a basis for making defendable predictions of the 
probability of technical critical values to occur in a system. 

3.2 Technical Sessions 

3.2.1 Session 1 - Uncertainty Identification and Quantification 1 

Paper 1- R. Childs, P. Reisenthel: Probabilistic Error Modeling in Computational Fluid Dynamics 

The work reported uses error modelling to quantify the inaccuracy in CFD results. It focuses on the error 
sources and treats error propagation either in a deterministic manner or by using a Monte Carlo approach. The 
method has been applied to four solvers, both structured and unstructured. For the authors, error modelling 
involves the solution of equations similar to the governing equations but for a forcing term due to the error 
source. The error sources convect and diffuse and may amplify or decay. The authors have found the 
following details of the error modelling process to be the most relevant: (a) truncation error at shocks, (b) 
Monte Carlo convergence acceleration, and (c) reduced forcing to help with stability. It is pointed out that 
truncation error calculation at metric discontinuities remains a key research challenge. 


Paper 2- M. Mignolet, P. Chen: Aeroelastic Analyses with Uncertainty in Structural Properties 

This paper focuses on uncertainties arising from the modelling of the real structure (model uncertainty) and/or 
from variability in the structural properties from one aircraft to another (data variability). The method used is 
nonparametric stochastic modelling. Two analyses are carried out: (1) flutter for the Goland wing with 
uncertainty in either the structural properties or its boundary conditions and (2) flutter for uncertain panels in 
supersonic flow. The authors point out that the fundamental problem of the nonparametric approach is the 
simulation of random symmetric positive definite real matrices and that the problem is also central to the 
introduction of uncertainty in nonlinear geometric structural models. The authors discuss distributing the 
matrix elements such that the maximum of the statistical entropy is achieved under the stated constraints of 
symmetry, positive definiteness, known mean model, non-singularity of the element matrix, and known 
measure of variability. For the first application, the authors demonstrate that the random stiffness coefficient 
coupling the first two modes of the design wing could dramatically affect the flutter boundary even at low 
levels of uncertainty. The second application showed similar sensitivity. 


Paper 17- B. Kleb: Toward Scientific Numerical Modeling 

The author points out that without known input parameter uncertainties, model sensitivities are all that can be 
determined and with code verification, output uncertainties are simply not reliable. To address these 
shortcomings, two proposals are presented: (1) an unobtrusive mechanism to document input parameter 
uncertainties in situ and (2) an adaptation of the Scientific Method to numerical model development and 
deployment. The first is addressed with the author’s method of uncertainty markup and the second with the 
author’s plea for component tests during software development. 
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Paper 4- B. Eussen, et al: Parameter Study on Dynamic Aeroelastic Simulations of Fighter Aircraft 

The authors carry out flutter analyses for a fighter aircraft in a “heavy store” configuration. They compare 
results from the linear doublet lattice method and inviscid CFD. They also consider other nonlinear effects: (1) 
oscillation amplitude, (2) Mach number, (3) initial angle of attack, and (4) initial trim state. They conclude the 
nonlinear effects are mostly important for the symmetric flutter modes and that the oscillation amplitude is 
more important than the initial state. 


Paper 5- V. Schulz, C. Schillings: On the Nature and Treatment of Uncertainties in Aerodynamic Design 

The authors suggest that the full potential of mathematical optimization can only be exploited if optimal 
designs can be computed which are robust to small (or even large) perturbations of the setpoint conditions. By 
that they mean that optimal designs should still be good designs even if the input parameters for the 
optimization problem formulation are changed by non-negligible amounts. They also point out that most of 
the techniques developed so far involve little or no nonlinearity. They provide some insight into the sources 
and ranges of uncertainties in the design process. They also give two approaches for doing this. 


Paper 6- T. Evans, et al: Identification and Quantification of Uncertainty Sources in Aircraft-Related CFD 
Computations - An Industrial Perspective 

The authors assess the major sources of uncertainty for aircraft flows and outline a possible way to incoiporate 
them into an MDO process. 

3.2.2 Session 2 - Code Verification 

Paper 7- L. Eca, M. Hoekstra: Code Verification and Verification of Calculations with RANS Solvers 

Based on the need for error estimation of complex RANS flow solutions, the paper addresses iterative errors, 
discretization errors and their interactions. The method of manufactured solutions is used to assess the mean 
flow and turbulence equations. While mean flow convergence plots display successfully the asymptotic range, 
questions arise about the role that nonlinear terms in the turbulence model play in convergence. The role of 
iterative errors on monotonic convergence is also covered. Based on the definition of numerical error, iterative 
error, and the discretization error, suitable estimates of the total error are based on the results of manufactured 
solutions. 

The ability of RANS solutions to reach the asymptotic range is demonstrated for flat plate flows and various 
turbulence models. As an example of large scale computation, the flow around a ship hull is presented. The 
integral coefficients of friction are found in the asymptotic range whereas the pressure drag is not, indicating 
insufficient geometry representation. 


Paper 8- A. Hay, et al: Adaptivity, Sensitivities and Uncertainties in CFD 

The presented method solves the RANS equations along with the sensitivity equations with respect to any 
shape parameter using classical finite element method, yielding formal second order accuracy for u, v, k, e and 
first order accuracy for p. The adaptive re-meshing is based on the Zhu-Zienkiewicz error estimator, which 
yields elemental error estimates. The Wiber error estimator is used as well for final estimation of errors. Code 
verification is performed by manufactured solutions to mimic boundary layer behaviour. Grid adaptation is 
verified using global error norms, order analysis, and point wise observations and the quality (order of 
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magnitude) of error estimate is verified as well. 

Code applications to the backward facing step are then presented and solution verification is investigated. This 
case indicates error estimates on the recirculation length, friction and pressure drag. The flow over a square 
obstacle mounted above a flat plate serves as a test bed for solution validation and uncertainty analysis. Using 
the sensitivity equation solutions, error bars due to uncertainties on the gap size are successfully computed. 

This paper appears to be a very good demonstration of the full chain of computational uncertainties that spans 
from code verification, solution verification using suitable error estimates, toward computational uncertainty 
analysis. 


Paper 9- C. van Dam, D. Chao: Wake-Based Aerodynamic Force Evaluation for CFD Verification and Drag 
Decomposition 

Accurate drag prediction is needed for aircraft drag reduction work and wake-based drag analysis can give 
valuable information about sources of drag. The paper presents drag equations along with the numerical 
approach to solve the RANS equations based on standard turbulence modelling. Numerical solutions for 
airfoils are analysed for sources of wave drag and viscous drag and the so-called higher-order terms that 
appear in the drag integral. 3D investigations are performed using a tapered wing. Drag components are 
displayed and the role of low-speed preconditioning for accurate drag extraction is emphasized. Results of 
lifting-line theory are shown for comparison. 

Questions remain open about the numerical uncertainties encountered in the present applications of wake drag 
extraction and the role that spurious entropy production plays. 


Paper 10- H. Xu: Development of Engineering Turbulence Capability with High Fidelity to Turbulence 
Physics using LES/DNS 

The paper aims to identify sources of errors in turbulence prediction with Detached Eddy Simulation (DES) 
and Direct Numerical Simulation (DNS) methods and to reduce these errors. The model errors in RANS are 
viewed in terms of the eddy viscosity concept. Turbulence in an annular duct is taken as a study case to 
illustrate the importance of turbulence anisotropy. One option to overcome this problem is grid-resolved Large 
Eddy Simulation (LES). For this purpose solutions of two grid densities are analysed with respect to their 
universal behaviour in the near-wall regions. The role of the iterative errors within the solution process is 
discussed in the context of multigrid iterative schemes. The importance of specifying boundary conditions is 
discussed and computational strategies of determining realistic boundary conditions are outlined. 

It is felt that the paper outlines useful guidelines to perform turbulent simulations of internal flows. However, 
it does not present any means to obtain quantitative uncertainty data from the calculation. 


Paper 11- R. Hixon, B. Anderson: Verification of Unsteady CFD Codes using the Method of Manufactured 
Solutions 

The paper is motivated by the need to provide high-accuracy unsteady solutions for computational aero- 
acoustics. The approach is to provide an external tool that needs no access to the flow solver source code. The 
tool is based on the method of manufactured solutions. The problem is that the source terms of the 
manufactured solutions have to be implemented into the code of interest. Therefore an external verification 
approach is devised here, where the manufactured solution is advanced in time by the external tool by one 
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time step and the result is compared to the discrete solution of interest. 

This concept is evaluated for two existing Computational AeroAcoustic (CAA) codes that use first-order 
implicit and explicit time stepping schemes. Error norms are evaluated to yield orders of convergence of 
various spatial discretizations and the results indicate consistent results. As it stands the tool can only test if 
the time-marching scheme of the original code works correctly. Work on an improved scheme is underway 
that is based on Taylor-expansion of the temporal behaviour in order to overcome these deficiencies. 

During the discussions it was acknowledged that the original Method of Manufactured Solutions (MMS) 
approach is able to identify coding errors term by term whereas the present form cannot be used that way. 

3.2.3 Session 3 - Uncertainty Identification and Qualification 2 

Paper 13- A. Hutton: Knowledge-based Procedures for Managing Uncertainty in the Industrial Application of 
CFD 

The paper presents the results of a network-based effort undertaken by the ERCOFTAC Special Interest 
Group. Two 3D test cases presented here highlight the numerical scatter of results, encountered by individual 
experienced users. These occur particularly due to interaction of errors, both model errors and errors related to 
numerical discretization. The ERCOFTAC first phase initiative is aimed to improve quality and trust in 
industrial CFD by establishing best practice guidelines as a published document. 

In its second phase, the QNET-CFD Knowledge Base (KB) was to be established. Based on six application 
areas and its corresponding flow regimes and application test cases, best practise guides for these application 
challenges were collected and located in a web-based KB to make it available to users. The data of the KB 
passed rigorous quality checks. As a result QNET-CFD has pulled together 10 years of testing and evaluation 
in a pragmatic way. Procedures proposed allow quality ensurance of test cases and solution evaluation. 
Flowever, the KB is changing continuously, as the level of physical models and the amount of numerical data 
encountered changes as well. ERCOFTAC will maintain the knowledge base and make it available 
worldwide. 


Paper 14- D. Bose, et al: The Use of Probablistic Modelling for Re-entry Thermal Protection System Design 

The paper is motivated by the difficulty to reproduce extreme environments of planetary re-entry along with 
the problem to define physical models for fluid and heat shield response. Simulations define flight 
environment data, material response models, test facility requirements and finally, ground-to-flight 
extrapolation. The sources of uncertainties include structural uncertainties due to lack of knowledge of 
physical models, uncertainties on large number of input parameters, and parameters due to non-perfect reality. 

Monte Carlo uncertainty analysis aims to obtain uncertainty margins with computed probabilities. The paper 
proposes the use of Latin Flypercube Sampling (LFIS) input parameter sampling rather than the traditional 
random sampling in order to use computational resources more efficiently. Applications to the Pathfinder 
mission to Mars using selected input parameters demonstrate that the Gaussian distribution is obtained with 
much smaller sample numbers, and the input mean is reproduced much better. The output mean and its 
standard deviation are significantly improved as well. This is important for uncertainty analysis where the 
uncertainty drivers are to be identified based on finite sample numbers. The findings open room for using 
larger coupled simulations within the computational design process. 
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Paper 15- D. Moens, D. Vandepitte: Internal Uncertainty Quantification in Numerical Models using Dynamic 
Fuzzy Finite Element Analysis 

The motivation of the paper is virtual prototyping in order to arrive at designs that comply with requirements 
and include uncertainties in the numerical model in terms of probability numbers. The present approach 
introduces non-determination into the model by using Fuzzy representations of functions. This is used to 
define membership functions on the input side to generate fuzzy output. The Fuzzy Finite Element method is 
based on interval analysis. The methodology to obtain output ranges taken from measurement data is 
described using frequency response functions FRF based on superposition of certain bounds on modal 
parameters. Uncertainty on structural damping is included and the algorithm ensures conservation. The 
computational algorithm has been scrutinized for efficiency and robustness of the inherent optimization tasks. 
Applications of rocket structure dynamics with multiple modes display the use of membership degrees in 
defining suitable uncertainties. 


Paper 16- W. Luber, et al: Definition Frequency of Exceedances of Structural Design Parameters 

The Flammer Shock (FIS) of a supersonic aircraft occurs due to a sudden flow expansion ahead of the 
combustion camber resulting in a strong shock wave running upstream into the intake. Shock strengths above 
a critical value cause plastic deformation of the intake structure. The FIS may only occur a few times during 
life time of the aircraft. Therefore FIS probability analysis and prediction of its strength are needed. The 
analysis of the present paper is built on Poisson functions of the occurrence of FIS and on probabilities of 
temperatures and FIS intensity, and aircraft mission characteristics are also included. The analysis yields 
probabilities for FIS strengths to occur and numbers of exceedances during aircraft life time. The results are 
comparable to the data of corresponding Monte Carlo simulations. The method proves useful to predict 
occurrence of FIS pressure exceedings during fleet life time. 


Paper 18- M. Mendenhall, et al: A Software System for the Preservation and Application of Best Practices for 
CFD 

The software system presented in the paper is motivated by the need to save cost and improve quality of CFD 
computations. It is well known that users play a critical role in the quality of CFD results. The BPX software 
systems aims at achieving best outcome in the application of CFD tools, particularly for uncertainty reduction, 
low error rates and to preserve corporate knowledge. The framework contains knowledge, practises and 
guidelines for specific codes along with background references. Its content has been accumulated from 
interviews and publications. The information is searchable by keywords and includes a tutorial on verification 
and validation. Grid examples are given and specific information including FAQs are available for the 
OVERFLOW Code. The paper includes samples of BPX usage that give a flavour of the type of information 
available. 


Paper 19- A. Probst, et al: Investigation of Computational Uncertainties of Airfoil Flow Phenomena Close to 
Trailing Edges. 

The present investigations are motivated by the special importance that the flow near trailing edges plays on 
lift and drag of the airfoil. Flow simulations investigated here are performed with the DLR TAU code and 
using hybrid coordinate grids. The results are presented for generic test cases of subsonic airfoils. Comparing 
c-grid and o-grid topologies it is found that an o-grid cannot be used to yield grid converged flow solutions 
near the trailing edge, even if large numbers of grid points are used. 
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The simulation sensitivities to geometry modifications of the trailing edge are investigated in the next step and 
significant differences occurred in the results pointing to the weakness of the standard Spalart-Allmaras 
turbulence model as comparisons with experimental data are considered. The effect of turbulence modelling 
on trailing edge flow is investigated for a generic airfoil with hinged flap. It is found that the prediction of 
separation onset at the trailing edge is most difficult to predict with uncertainties in the corresponding angle of 
attack of about 2 deg. This holds even for well calibrated second-order closure turbulence models. 

The paper demonstrates a significant amount of computational uncertainties related to trailing edge flows even 
for state-of-the-art turbulence models and the need for detailed investigations into the various possible 
sources. 

3.2.4 Session 4 - Code Validation 1 

Paper 20- E. Tinoco: CFD Uncertainty and Validation for Commercial Aircraft Applications 

The author points out that the use of CFD in aircraft design is now crucial for commercial success. For 
example, the Sonic Cruiser program would have been inconceivable without the use of CFD. The paper 
presents the processes and techniques used at the Boeing Company to create consistency in the CFD results so 
that the uncertainties can be confidently bounded and their impact on design decisions limited. He stressed 
that success is attained by understanding the impact of the uncertainties on the final product. He presents 
many case studies from development programs for subsonic transport aircraft. In his concluding remarks, he 
points out that getting to the stage where CFD is used with confidence in a predictive manner to make 
development decisions is a long and costly endeavour, frequently costing more that the original code 
development. He states that the reason is that it is not just the code that must be validated for its intended 
purpose, but also the entire process of geometry, grid generation, solver, post-processing of results, and even 
the user that must be verified and validated. 


Paper 21- F. Stern: Quantitative V&V of CFD Solutions and Certification of CFD Codes 

The author presents his approach for quantitative assessment of numerical and modelling errors and 
uncertainties for CFD simulations and of intervals of certification for CFD codes. Examples are provided for 
ship hydrodynamics and critiques of other approaches are offered. The author points out that the success of 
the methodology depends strongly on reaching the asymptotic region of grid convergence. 


Paper 22- S. Melber-Wilkending, et al: CFD Validation by Wind Tunnel Measurements: Uncertainty 
Assessment by Numerical Simulation of Complete Wind Tunnel Flows 

This paper proposes to investigate CFD uncertainties by considering the complete numerical simulation and 
analysis of wind tunnel experiments including all geometrical and aerodynamic conditions, in effect 
generating a numerical wind tunnel. This is in stark contrast to the usual research and project approaches of 
using wall models to correct the wind tunnel results to free flight conditions and compare with CFD, also done 
for free flight. To quote the author, “From the deviations detected by careful comparisons of the experimental 
data with the results of the numerical simulation of the experiment, correction rules will be derived.” 
However, it is not clear to the reviewer how the uncertainties of the rules would be derived. 
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Paper 24- F. Davoudzadeh: Validation of Computational Fluid Dynamics Simulations for Realistic Flows 

The author describes strategies used to verify and validate CFD calculations via case studies of realistic flow 
simulations and presents interesting comparisons with exact solutions and experiment. The primary effort was 
aimed at achieving reasonable vortex flow development (without excessive dissipation) and using that model 
to simulated complex flows such as those for submarine crashback and depth-changing maneuvers. 


Paper 26- B. Thacker, et al: Model Validation & Uncertainty Quantification Applied to Cervical Spine Injury 
Assessment 

The authors review the ASME model verification and validation process and apply it to the development and 
validation of a complex cervical spine model for assessing potential injury during aviator ejection or evasive 
maneuvers. Four hierarchical levels were used for the spine model. The component (lowest) level consisted of 
individual tissue properties for the ligament models which were calibrated with experimental data. Level 2 is 
the intervertebral disc construct, level 3 a complete motion segment, and level 4 the full cervical spine 
column. The uncertainties from the calibrations of the level 1 models were propagated into the upper levels 
and the results for the upper levels were compared to experimental data in the context of the uncertainties. The 
reviewers recommend the approach presented in this paper and encourage others to read it as a case study of 
proper code validation. 


3.2.5 Session 5 - Numerical Accuracy 1 

Paper 41- D. Drikakis, et al: Computational Uncertainty in CFD Associated with Spatial and Temporal 
Discretization and Non-linear Methods Design 

The paper describes the results of high-resolution methods in compressible flow simulations. The aim is to 
reduce numerical uncertainties in vehicle design. The approach is to use higher-order methods with implicitly 
modelled non-resolved turbulent scales for compressible flows. Based on a review of existing higher-order 
schemes the study cases of an airfoil, a swept wing and a transonic cavity are used to demonstrate the relative 
merits of various higher-order schemes for LES simulations. Implicit turbulence modelling is preferred for 
compressible flows as being less diffusive, self adjusting and numerically robust. 

Helmholtz and Richtmeyer-Meshkov instabilities are defined as generic test cases for turbulent mixing. 
Significant improvements are reported for so-called monotone methods in terms of spatial and temporal 
resolution. However, the improved resolutions some times are not exhibited in global growth rates of the 
mixing process. 


Paper 27- C. Roy, A. Sinclair: On the Generation of Exact Solutions for Evaluating Numerical Schemes and 
Estimating Discretization Error. 

The Method of Nearby Problems proposed in the paper is motivated by the need to provide exact solutions of 
realistic problems for both code verification, discretization error estimates and mesh adaptation. The method 
builds upon the ability to reconstruct smooth analytic solutions from discrete numerical data using spline fits. 
Extension to multiple dimensions is performed by a series of local fits with proper weighting functions. 

The generic 2D heat conduction problem is chosen to demonstrate the potential of the nearby solution 
approach in comparison to the known analytical solution to this specific problem. It is seen that the spline 
fitting error converges with grid refinement. The analytic nearby solution is then used to assess numerical 
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errors of the discrete problem. The 2D lid-driven cavity is taken as a more complex and realistic application. It 
is demonstrated that the local spline fitting approach is essential to create a sufficiently close nearby problem. 
Good agreement is reported for the comparison of errors for this flow problem provided the spline fitting error 
is smaller than the discretization errors. 

The approach is viewed as very promising strategy for future verification of compressible and turbulent 
computations, however, it would need extension to unstructured grids. 


Paper 28- S. Borel-Sandou, G. Roge: Improvement of Observations, Error Estimation and Mesh Adaptation 
using an Adjoint Based Approach for Aerodynamic Calculations 

The error estimation work presented is motivated by the need to accurately simulate flow around of aircraft. 
The approach is to solve the adjoint problem originated by Miller and Giles for defining error bounds and 
output-based mesh adaptation. This adjoint approach approximates the problem to compute an estimation of 
the error on a fine grid without solving the flow problem there. Confidence intervals for the error are also 
derived. Code verification for transonic and supersonic applications is assessed by solving the ramp flow 
problem and the flow around an airfoil. Supersonic inviscid flow over a business jet is defined as a realistic 
and complex test case. The near field pressure signature of the flow is needed to predict sonic boom 
characteristics on the ground. The adaptive flow solutions presented by the present approach agree reasonably 
well with other published solutions, however, questions remain about the relative efficiency of the present 
approach. 

Paper 29- V. Couallier, M. Delbos: Accurate Boundary > Conditions with Classical and Higher Order CFD 
Methods 

Based on a review of CFD modelling approaches, application areas and capabilities the paper aims to improve 
one important aspect of numerical computation, that are numerical methods for treating boundaries of the 
computational domain. Richardson extrapolation and the Grid Convergence Index are introduced to evaluate 
the numerical error. Analysis is performed on a simple generic airfoil for inviscid flow. Various procedures of 
error estimation are presented along with variations of the boundary algorithm. Similar investigations are 
presented for viscous flows. Boundary treatment is found to be rather important for acoustic simulations 
where characteristic boundary conditions are needed to avoid non-physical reflections. 


Paper 30- D. Veley, J. Camberos: Computational Uncertainty with Reconfigurable Computing 

Reconfigurable computing is motivated by the fact that continuous speed up of processor units has come to an 
end. Field programmable gate arrays, on the other hand, allow customized, integrated circuits. The 
performance of these units has greatly improved thanks to parallelization. Bit reduction offers good 
perspectives for further speed up. Based on detailed description of binary representation, basic calculation 
operations and simple iterative schemes the effects of bit reductions on numerical accuracy are analysed. It is 
seen that bit demands increase with refining the grid. It is therefore recommended to couple bit refinement 
with multigrid techniques to reduce computation time. 


Paper 32- 1. Celik, et al: A Reliable Error Estimation Technique for CFD Applications 

The paper suggests a variant of the Approximate Error Spline Method (AES) as opposed to Richardson 
Extrapolation. Here the approximate error is assumed proportional to the true error. The local error can then 
be approximated from data taken from various grid triplets. The concept is successfully tested for one- 
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dimensional scalar transport and two-dimensional scalar transport with convection and diffusion where 
manufactured solutions are used for defining the exact solutions. Manufactured boundary-layer type solutions 
are employed for the 2D Navier-Stokes solver and the approximated error is found in close agreement with the 
true error. The paper devises manufactured solutions for the backward facing step problem using the stream 
function approach. These are then used for computations using the Fluent code. Again, the agreement of 
approximated and true errors compare quite well, using the proper computed proportionality constant. 

3.2.6 Session 6 - Propagation Methods 1 

Paper 33- C. Hirsch, C. Dinescu: NODESIM-CFD: A European Project on N on-Deterministic Simulation for 
CFD-Based Design Methodologies 

The authors give an overview of the NODESIM-CFD project which intends to incorporated operational, 
geometrical and numerical uncertainties in the simulation process. The overview presents the objectives of the 
project together with the tasks being undertaken and some preliminary results. Some of those results are 
presented in the present symposium papers 6, 28, 29, 34, 35, 36, 46, 48, 63. The paper points to work being 
done on measurement and statistical characterization of input uncertainties. This is in contrast to much of the 
work being done elsewhere using simple guestimates of the ranges of input uncertainties. At the symposium, 
the results of statistical characterizations of blade roughness measurements for four different manufacturing 
methods were presented as examples of the methodology. Those results, not in the paper, will be published in 
the future. 


Paper 34- J. Witteveen, et al: Long-Term Stochastic Behavior of Aeroelastic Systems - Development of a 
Robust and Efficient Adaptive Stochastic Finite Element Method 

In this paper, the authors attempt to overcome the problems of sensitivity to small input variations for time- 
dependent problems. They propose a formulation which maintains an approximately constant accuracy in time 
with a constant number of samples. This is accomplished by using a time-independent parameterization of the 
sampled time series in terms of frequency, phase, amplitude, reference value, damping, and higher-period 
shape function. This parameterization is interpolated using a robust adaptive stochastic finite elements method 
based on Newton-Cotes quadrature in simplex elements. The effectiveness of the approach is demonstrated 
with two unit problems and verified by comparison to Monte Carlo results. 


Paper 35- G. Loeven, et al: Radial Basis Functions for Uncertainty Quantification in CFD - A Preliminary 
Study 

The probabilistic radial basis function approach is applied to several unit problems with multiple uncertain 
parameters to gain some insight into its application as an efficient uncertainty quantification method. The 
authors mainly studied (1) convergence of the error of the mean and variance of the solution, (2) uniformity of 
the samples distribution in probability space, and (3) the ability of adding an extra sample. 


Paper 36- L. Parussini, V. Pediroda: Ficticious Domain with Least-Squares Spectral Method to Explore 
Geometrical Uncertainties by Chaos Collocation 

The authors apply their approach to the study of the effects of geometric tolerances for one- and two- 
dimensional elliptic problems to demonstrate the accuracy and convergence of the method. They use a non- 
intrusive approach which does not require modification of the legacy code. Their approach also avoids 
remeshing. They point to further research needed (1) to compare the advantages and disadvantages of 
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intrusive and non-intrusive methods, (2) to develop effective criteria for choosing collocation points, and (3) 
to validate the method beyond the advection-diffusion equation. 


Paper 37- U. Tabak, et al: Combined Uncertainty Effects on the Dynamic Behavior of Mechanical Structures 
Based on the Fuzzy Element Method 

The authors describe a fuzzy finite element method for handling uncertainties in an efficient framework and 
apply the method to the frequency response analysis of a satellite baffle. They show that the entire process of 
mesh moiphing, finite element analysis, fuzzy finite element analysis is efficiently integrated. 


Paper 38- F. Poirion: Uncertainty and Aeroelasticity: ONERA ’s Experience Review 

The author describes several ONERA case studies of analysis of uncertainty effects on the stability of fluid- 
structure coupled systems. The paper focuses on various models for introducing uncertainty in an elementary 
stiffness matrix and shows that a polynomial chaos representation whose coefficients are estimated through 
Monte Carlo simulation yields a better approximation for the matrix and its eigenvalues than the usual Taylor 
series expansion. 

3.2.7 Session 7 - Numerical Accuracy 2 

Paper 39- R. Cummings, et al: Experiences in Accurately Predicting Time-Dependant Flows 

Based on a review on predictions of vortices on military vehicles the authors emphasize the importance of 
choosing the time steps and grid densities needed to resolve unsteady vortical flows. The phenomena of 
unsteady vortex motions on a delta wing are described as a physical basis for refinement. Detailed 
investigation into the problem are presented for three generic test cases, unsteady flow over a low Reynolds 
number 2D airfoil, and unsteady laminar and turbulent flows over a delta wing. The interactions of grid and 
temporal resolution to capture vortex shedding are presented for the airfoil. Investigations on the delta wing 
focus on the helical vortex motion and the vortex breakdown along with resolving the turbulent energy in the 
vortex core. Unsteady flows over the F-18C aircraft are then scrutinized. It is found that the frequencies of the 
vertical tail pressure fluctuations compare very well. For computations of the F-16XL experiences in selecting 
appropriate time steps for resolution of multiple frequencies of interest are reported. However, no useful 
method for numerically adapting the time step to the flow has been identified so far. 


Paper 40- H. Ozdemir, H. Hoeijmakers: Higher-Order Discontinuous Galerkin Method for Aeroacoustic 
Wave Propagation on Distorted Grids 

The proposed method solves the linearized Euler equations with acoustic source terms. The underlying 
Galerkin method is based on the assumption of parallel hexahedral sides. The Riemann problem is solved by 
the Lax-Friedrich flux. The scheme does reproduce the theoretical orders of accuracy for smooth meshes. The 
effects of distorted grids are investigated for the generic case of acoustic radiation from a vibrating wall 
segment. The effect of skewed grids is displayed for various accuracy orders of the scheme. Randomly 
distorted grids, however, yield larger errors into the solution. A further test case addresses the resolution of a 
slope discontinuity in the incoming waves of an acoustic liner that are diffracted at a slit. 
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Paper 42- S. Esquieu: Reliable Drag Extraction from Numerical Solutions: Elimination of Spurious Drag 

The paper aims to analyse numerical errors in drag computation and their sources. Direct integrations of drag 
values on the wing surface are sensitive to numerical discretization errors. Drag equations based on wake 
integration can be cast to discriminate between physical and spurious drag sources. Drag breakdown of these 
components is displayed for a wing-body configuration. More detailed analysis on the source of spurious 
pressure drag and the effect of refining the grid is presented for inviscid flow over a transonic airfoil. The 
importance of controlling spurious drag is displayed for patched structured grid discretizations around the 
fuselage nose. An important source of spurious drag in farfield drag extraction methods is the diffusion of the 
wing tip vortex behind the wing. This drag needs to be corrected by the spurious drag sensor proposed here. 
The approach is seen to be very valuable in improving the numerical accuracy of drag prediction. 


Paper 44- G. Deng: A Solution Qualification Procedure Applied to a Backward Facing Step Test Case 

The focus of the paper is on error norms and the ability of achieving the expected orders of accuracy. Well 
established extrapolation methods such as Richardson’s often fail in real test cases. The present work employs 
a finite-volume unstructured-grid method for incompressible flow. Code verification with manufactured 
solutions indicates second order accuracy for the inviscid fluxes, whereas the viscous terms and the source 
terms display first order errors for abrupt changes in mesh size. Assuming a monotone variation of errors from 
the finest grid to the coarsest the authors extrapolate the error of the finest grid using different extrapolation 
variants. This procedure is checked with manufactured solutions and then applied to the solution of flow over 
a backward facing step. Here, different grid topologies are chosen for testing error extrapolation variants. The 
extrapolation approach allows for defining local error bars of the flow solution. 


Paper 45- R. Figliola, J. Camberos: Numerical Uncertainty Estimation in Aerodynamic Drag and EM Scatter 
Computations 

Existing flow calculation methods are investigated to analyse uncertainty in drag prediction for supersonic 
body and subsonic wing flows. The calculations of the conical fi nn ed cylinder involve solution of the Euler 
and the Maxwell equations. Grid effects are investigated by using a family of four grids. Grid refinement 
index is used to achieve 5% uncertainty levels on drag. Similarly, uncertainties on wave amplitudes are 
obtained for various grid densities. The wing drag study is performed for a generic, unswept wing and 
computed data is compared to lifting line theory. The authors report on grid density studies of the numerical 
drag computation and on the effect of numerical dissipation. The individual sources of errors identified by 
numerical experiments are then input to a global error estimation. Finally, these total drag uncertainties are 
explained as reasonable and useful. 


Paper 46- R. Dwight: A Posteriori Error Estimation and Mesh Adaptation for Finite Volume Methods 

The paper focuses on perturbation methods to analyse discretization errors. It is motivated by the observation 
that local gradient adaption methods are not reliable for accurate flow field predictions. The theory of error 
estimation builds on the desire to find a better approximation to the local error as given by the actual 
discretization on a given mesh. It is explained that the error can be represented in terms of the solution of the 
adjoint problem. In order to avoid inefficient residual calculations on finer meshes, one may take the 
dissipative terms present in the numerical solutions as an estimator of the error on the given mesh. This is then 
used in the adjoint scheme for feature-based grid adaption. The new adaptive method is seen to improve 
accuracy for lift and drag for a wide range of inviscid flows. It can be seen that this adaption works reliably 
and it is much more efficient than global refinement. For viscous flows it has been shown that the method 
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estimates the numerical error in the pressure distribution very well. 

3.2.8 Session 8 - Propagation Methods 2 

Paper 47- S. Hosder, R. Walters: Non-intrusive Polynomial Chaos Methods for Stochastic CFD - Theoiy and 
Applications 

The authors present the theory for non-intrusive polynomial chaos used for uncertainty propagation with an 
emphasis on their point-collocation technique. The method is demonstrated with two examples: (1) stochastic 
expansion wave problem with an uncertain deflection angle and (2) stochastic transonic wing with uncertainty 
Mach number and angle of attack. They show that their method promises considerable savings in resources 
while maintaining acceptable accuracy for the resulting statistics. 


Paper 48- C. Lacor, S. Smirnov: Uncertainty Propagation in the Solution of Compressible Navier-Stokes 
Equations using Polynomial Chaos Decomposition 

The authors present an intrusive polynomial chaos method for error propagation for the compressible Navier- 
Stokes equations. The results are applied to two test problems: (1) quasi- ID Euler flow in a nozzle with 
uncertain inlet conditions and (2) 2-D laminar lid-driven cavity flow with uncertain viscosity. The polynomial 
chaos results are compared with Monte Carlo results. 


Paper 49- X. Zhu, et al: Probabilistic Uncertainty Analysis: High Performance Computing Aspect 

The authors tackle the problem of efficiently carrying out uncertainty analyses for partial differentiation 
simulations of large-scale randomly heterogeneous linear dynamic systems using Latin hypercube sampling. 


Paper 50- N. Lindsley, P. Beran: Methods for Quantifying Uncertainties in Aeroelastic Responses 

The authors use reduced order modelling to accelerate the stochastic analysis of an aeroelastic system 
consisting of a clamped square plate in high-speed flow. The reduced order model is developed using 
orthogonal decomposition. The results compared with the full-order model show that the method provides two 
orders of magnitude reduction in cost while giving a good approximation of the response statistics. 


Paper 52- C. Burdyshaw, W. Anderson: Advances in Discrete Sensitivity Methods Applied to Uncertainty 
Analysis 

The authors address developments in computational sensitivity analysis which use a suite of discrete methods 
(direct and adjoint) to improve accuracy, efficiency, implementation and extensibility. 


Paper 53- L. Hinke, et al: Propagation of Uncertainty in Structural Dynamics using Component Mode 
Synthesis Methods 

The authors propose that component mode synthesis offers an appealing framework for the analysis of 
structural dynamics of uncertain structures, particularly with regard to the way uncertainty is included, 
quantified and propagated. The paper explains in detail why the method works and includes discussion of the 
ease of implementation of uncertainties. 
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Paper 54- S. Turrin, M. Hanss: Uncertainty Analysis in Crash Simulation with Respect to Structural Design 

The authors show the dominant character of the initial and boundary conditions (velocity and impact angle) of 
a crash compared to the material and geometrical parameters of the structure. They also show that tiny 
changes in those conditions can lead to enormous variations in the structural response, begging the question 
then of conventional crash analysis. They conclude that, in general, the uncertainty propagation is strongly 
dependent on the structural design and, hence, the effect of the uncertainties can be successfully reduced by 
developing designs that are robust to those uncertainties. 

3.2.9 Session 9 - Code Validation 2 

Paper 55- J. Morrison, M. Hemsch: Statistical Analysis of the AIAA Drag Prediction Workshop CFD 
Solutions. 

The Drag Prediction Workshops (DPW) aim at assessing the state of the art for the prediction of forces on 
industry relevant geometries. The participants choose joint 3D test cases on wing-body configurations and 
isolated wings. In the third DPW participants used 14 CFD codes. The statistical analysis includes results 
with grid refinements for two configurations. So-called outliers have been identified in a systematic way. 
Scatter in solutions include modelling error, process error, code error, numerical error. Once the outliers are 
removed from the data set one observes a rather collective convergence of drag values with grid density. 
Adversely to the expectations, this trend could not be observed for the wing-alone cases. 

DPW-3 was designed as a blind test and evidence of grid convergence of the wing-body could be achieved. 
However, the scatter of numerical results as compared to DPW-2 could not be much reduced. No clear reason 
for the remaining discrepancies could be identified based on statistical relevance. Grid generation still remains 
a challenge, especially if the task is to generate families of grids. A fourth DPW will be held in Summer 2009. 


Paper 56- O. Boelens: Feature-Based Code Validation using F-16XL Flight Test and Wind Tunnel Data 

The paper covers work to enhance confidence in CFD methods as applied for complex fighter aircraft 
configurations. F-16XL exhibits a cranked arrow wing and it was equipped with surface pressure tabs, 
boundary layer rakes, and skin friction instrumentation. Simulations were performed with the structured finite- 
volume method ENSOLV. The grid contained about 14M points. The paper displays three computational 
cases which were computed assuming steady flow. The discrepancies in the pressure distribution as obtained 
for the subsonic cases are explained by the adverse effects of a locally coarse mesh and weakness of the 
turbulence model. A transonic case with Mach = 0.93 was also computed. Here, one finds larger deviations of 
the computed and measured pressure data. Investigations into possible sources of this discrepancy have not 
revealed any defendable explanation for this. Additional computations were done to be compared to wind 
tunnel data for both steady flow and unsteady flow due to pitching motion of the model. Reasonable 
agreement with the measured data is reported. 


Paper 57- J. Vassberg et al: Summary of the Third AIAA CFD Drag Prediction Workshop 

This paper addresses in detail some more specific flow phenomena of the joint wing-body configuration and it 
presents further data evaluations compared to Paper 55. The effect of wing body juncture separation on 
pressure distributions and near-wall flow is presented. The computations display a large variation of the 
spanwise extent of this separation between the solutions. Richardson extrapolation is used to evaluate the 
convergence data and a figure of merit is computed by assembling the deviations from ideal second-order 
behaviour. Using the figure of merit outliers are identified and this reduces the scatter of data considerably. 
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The results are discussed in terms of various drag polars Some of the scatter can possibly be traced to the 
choice of the grid, and partly to turbulence models and thin-layer approximations versus full viscous terms of 
the Navier-Stokes equations. It is concluded that the results generally show room for future improvements in 
drag prediction even if the scatter of drag data is now at about 2% (without the outliers). 


Paper 58- J. Gorski, et al: Uncertainties in the Resistance Prediction of Underwater Vehicles 

Based on the need to compute accurate resistance data of underwater vehicles, the paper deals with solution 
verification and model validation for generic streamlined bodies. Two flow solvers are presented, the code 
UNCLE for structured grids and the code TENASI for unstructured meshes. Solutions on structured grid and 
unstructured grids with varying density have been systematically analysed using both codes. Monotonic grid 
convergence of the numerical resistance coefficients to errors smaller than about 0.5% are observed with the 
TENASI code while UNCLE displayed still non-monotonic results. 

Validation investigations cover streamlined bodies of various fineness ratio and Reynolds numbers along with 
using three different turbulence models. While the results given can contribute to building confidence into 
RANS drag prediction methodologies for underwater body design it is felt that more high-quality 
experimental data with well characterized uncertainty are needed to obtain quantitative levels of 
computational uncertainty. 


Paper 59- D. Sornette, et al: Theory and Examples of a New Approach to Constructive Model Validation 

While solution verification is described in this paper as a rather well understood and established process, 
model validation is characterised as less defined and there exist basic so called “Impossibility Statements” 
about the chances of reducing complex natural systems. The main scope of the paper is then to establish a 
theory of constructive, iterative validation processes. One validation loop is here defined as a process that 
increases or decreases the trust coefficient, V, of a particular model. The trust measures depend on the 
probability, p, that the model matches the experiment, on the performance, q, of other existing models, and on 
the novelty and importance of the experimental data, c. Validation can be asymptotically satisfied if the 
number of loops and the final trust value are sufficiently high. The paper then describes some useful 
properties to obtain metrics of trust, based on p,q, and c and the need to rigorously consider computational and 
experimental uncertainties within this validation process. 

Two examples that use the proposed validation theory are finally presented to illustrate the nature of the 
process. It is concluded that the remaining aspect of the proposed theory for future refinement is to better 
evaluate the probability, q, and to incoiporate uncertainty quantification. 

3.2.10 Session 10 - Propagation Methods 3 

Paper 61- A. Martowicz, et al: Coupling MCS with RSM for Evaluating Dynamic Properties ofFE Model of 
Real-Life Vehicle ’s Structure 

The authors present an uncertainty analysis of the input parameters on a dynamic finite element model of the 
Fiat Punto. To reduce the order of the problem, metamodels and genetic algorithms were used to choose the 
most influential terms in the approximations — in effect a kind of sensitivity analysis. The authors show that, 
of the original 30 uncertain parameters, only 14 were necessary to effectively model the changes in the first 
natural frequency of the model’s vibration. 
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Paper 62- N. Lindsley, et al: A Multi-Fidelity Framework for Quantification of Uncertainty in Aeroelastic 
Systems 

The authors present a combined sensitivity and uncertainty-propagation analysis. They first conduct a 
sensitivity analysis of the baseline wing application to reduce the number of parameters. They then carry out a 
design optimization analysis to maximize the first natural frequency while constraining the rest. Monte Carlo 
simulation is then used to propagate the input variation. Flutter boundaries are estimated and it is shown that 
there is a strong sensitivity of the flutter speed to small changes in the structure with apparent switching in the 
failure modes. 


Paper 63- E. Herbin, Q. Dinh: Management of Uncertainties at the Level of Global Design 

The authors present a case study of the management of uncertainties during the design of an aircraft. The 
authors observe that the first-order second moment method uses a linearization of the function that relates the 
input variables and parameters to the output variables which can lead to problems when the mean value of the 
input variables is close to a local/global extremum. In such a case, the method computes a zero uncertainty 
because the first derivative of the function is close to zero. The authors point out, as have others, that a 
quadratic reconstruction in necessary in such cases. 


Paper 64- R. DeLoach: Propagation of Computational Uncertainty using the Modern Design of Experiments 

The author describes the use of formally designed experiments to aid in uncertainty propagation for a 
computational experiment. The underlying code is approximated with low-order polynomial graduating 
functions to the response function. A practical method is offered for quantifying the response surface error 
that can be attributed to imperfect knowledge of independent variable levels. The author presents the results of 
a case study involving propagation of uncertainty in space shuttle thermal and structural re-entry loads. 


4.0 REVIEW OF TECHNOLOGY STATUS 

4.1 Method Qualification 

4.1.1 Code verification 

Code verification is defined as the process to obtain evidence that the implementation of the mathematical 
model used for computation of numerical solutions is correct. This is generally a demanding objective since: 

• numerical simulations of dynamical systems involve complex spatial and temporal discretization 
schemes and often coupling algorithms between different solution domains. 

• exact and sufficiently meaningful solutions to the nonlinear equations governing fluid and structure 
dynamics are not available. 

A generally valid and technically feasible strategy to this overall problem is not known, and the symposium 
contributions focussed on practical approaches to demonstrate code verification for specific flow solvers 
applied to single solution domains. 

The Method of Manufactured Solutions defines analytical expressions tailored for specific flow phenomena 
that can be used as a replacement for the - not available - exact solutions. This approach is used in several 
papers, No. 7, 8, 1 1. In particular, the correct reduction of discretization and iterative errors within the solution 
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process has been assessed, and asymptotic convergence of RANS solutions could be demonstrated. The 
method is being extended for time-accurate flow solutions. 

A promising alternate approach was presented in Paper 27 where piecewise analytic solutions are generated 
from discrete numerical solutions via multidimensional spline fitting. This approach opens the way to 
verification for general flow models and complex flow fields. 

Note that rigorous code verification for multidisciplinary physics simulations was not covered by the 
presentations of the symposium. It is felt that code verification in this area represents an unresolved problem 
for computations with coupled legacy codes and can only be overcome with future monolithic solution 
approaches. 

4.1.2 Model validation 

Model validation is defined as the process of quantitative determination that the model is an accurate 
representation of the real world for its intended use. 

It is the authors’ experience that nearly all workers in this arena use graphical comparisons to qualitatively 
state validation of their models/methods with respect to reality. There is very little quantification using 
methods such as those described by Oberkampf in Paper P3, Thacker in Paper 26, and Somette in Paper 59, 
especially with regard to hierarchical validation of complex models and this conference was no exception. 
While direct comparisons are essential for knowledgeable workers to be able to determine if the physics are 
being captured adequately, they are usually not sufficient for estimating uncertainties associated with the 
model for predictive purposes. Detailed quantitative measures of the mismatch are required as well as some 
understanding of the physical reasons for the mismatch. Besides the validation methods papers (P3, 21, 26, 
59), there were several other papers which presented interesting qualitative comparisons of computational 
models/simulations with experiment: 8, 19-26, 56, 58, 59. 

4.2 Qualification of a Specific Prediction 
4.2.1 Solution verification 

Solution verification of the numerical solution usually represents the first step in the estimation of simulation- 
based predictive capabilities for a technical system. It deals with the evaluation of discretization and iterative 
errors of the solutions. 

The problem is that the true errors are usually not accessible in practical, predictive applications. The most 
widely used method of error estimation is by grid refinement computations and extrapolation of the data to the 
continuous limit. This approach was used by many contributors to the symposium in one way or another, i.e. 
Papers No. 7, 8, 10, 19, 21, 32, 39, 41, 42, 44, 45, 55, 57. However, it turns out that the well-known 
extrapolation process for this purpose originated by Richardson is often not very reliable and some 
alternative procedures were recently devised as presented in Papers No. 32 and 44. 

A drawback of estimating the errors by multiple grid refinement is the large associated cost which makes this 
approach impractical for routine applications, particularly in three dimensions. Several alternate approaches to 
this problem have therefore been presented during the symposium. One possibility is to use locally higher- 


3 M. D. Salas has addressed some issues associated with this observed lack of reliability (i- Computers & Fluids, 35 [2006] 688-692; 
ii- Computers & Fluids, accepted for publication). 
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order solution reconstructions as an error estimate (No. 1, 8). This is closely related to generating piecewise 
analytic Nearby Problems for error estimation, as advocated in Paper No 27. 

The need for computational efficiency encourages one to consider only those discretization errors that are 
relevant to the system quantity of interest. It turns out that these errors can be represented in terms of the 
solution of an adjoint problem. While the straight-forward application of the adjoint equation approach for 
error estimation would make residual computations on refined meshes necessary (see Papers No. 28 and 46), 
one can also formulate simplified versions that require less numerical effort. Paper 46 takes the dissipative 
terms present in the numerical solution as an estimator of the error on the given mesh. This is then used in the 
adjoint scheme for feature-based grid adaption. Impressive numerical gains against global grid refinements 
and good correlations of the estimated error versus accurate numerical error computations are reported. 

Note that a posteriori error estimates for unsteady flow solutions have not been covered by the contributions 
to the symposium and this area is seen as a challenge for future research. 

It is concluded that reliable numerical error estimates are a key requirement for truly predictive simulations 
and further progress with practical approaches in this area is needed. Future simulation systems for design and 
development of military vehicles should offer capabilities of providing quantitative numerical error estimates 
of predicted flow or structure quantities. 


4.2.2 Uncertainty analysis 

Uncertainty analysis is defined as the process of identification and characterization of uncertainty sources. 

The papers presented in this conference in this area attacked the characterization of a wide variety of 
uncertainty sources in a wide variety of ways: 

• (1) - discretization errors 

• (15) - modal analysis 

• (16) - hammershock loading 

• (17) - chemistry of high-speed external flows 

• (26) - properties of human spinal ligaments 

• (33) - blade roughness; presented at the conference but not included in paper. To be published. 

• (37) - fuzzy properties 

• (54) - initial and boundary conditions 

• (61) - material and geometrical properties 

4.2.3 Uncertainty propagation 

Uncertainty propagation is defined as the process of propagating the uncertainties into System Response 
Quantities (SRQs). 

The conference papers showed that this is a robust area of research. The following approaches were covered: 

• Monte Carlo (1, 14, 34, 50) 
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• Nonparametric Stochastic Modelling (2, 

• Latin Hypercube (14, 49) 

• Unsteady Adaptive Stochastic Finite Elements (34) 

• Probabilistic Radial Basis Functions (35) 

• Chaos Collocation (36) 

• Fuzzy Interval Analysis (37, 54) 

• Polynomial Chaos (38, 48) 

• Adjoint Sensitivities (52) 

• Response Surface (63, 64) 

Reduced order modelling seems to be one of the most viable methods presented for improving efficiency 
without giving up essential physics. The issues at the conference for other methods that are designed to 
improve efficiency are (1) retention of the essential physics and (2) the need to validate such approaches for 
each type of problem/physics. 

4.2.4 Sensitivity analysis 

Sensitivity analysis is defined as the process of determining the most important contributors of the uncertainty 
sources. 

The sensitivity analysis applications presented at the conference were: 

• Aeroelasticity (2, 4, 62) 

• Structural Dynamics (15, 54, 61) 

• CFD(8, 19,28) 

• Aero thermal Heating (14, 64) 

5.0 FINAL REMARKS AND OUTLOOK 

The reviewers found that several of the authors’ comments were particularly indicative of the state of the art 
and/or pointed to important considerations. A selection is presented below in italics together with additional 
co mm ents by the current reviewers: 

• Tinoco (20) - The use of CFD and therefore the need for [uncertainty quantification] has been driven 
by desperation. In his presentation, Tinoco pointed out that the desperation comes from intense 
competition pressure to open the design space as much as possible and wring out as much 
performance as possible. This point was also made by two of the plenary speakers, Raj (PI) and 
Shahpar (P2). 

• Raj (PI) - Computational uncertainties [are] the Achilles’ Heel of simulation-based design. Raj here 
is referring to the acceptance of CFD-based simulation being driven, particularly in complex cases, by 
the decision-maker’s ability to quantify risk, based, at least in part, on the estimated simulation 
uncertainties. 
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• Thacker (26) - We need to get the right answer for the right reasons. Thacker’s point here is that one 
can get the right answer in a calibration/validation exercise; but, if that answer is not based on a 
proper validation hierarchy and on sufficient data at each stage, there would be little confidence that a 
prediction would necessarily be accurate. 

• Somette, et al (59) - ... validation is concerned with the value of the model vis-a-vis experiment: it 
provides the evidence that may convince one to use its predictions to explore into new parameter 
regimes . Thus, it could be said that the quality of the validation results (and validation process) 
determines our confidence in our predictions made with the model. 

• Oberkampf (P4) — The [model] validation metric is the evidence for mismatch between experiment 
and simulation. This point cannot be overemphasized. It is almost always the case that users and 
method developers look for how well the method matches the data, usually doing graphical 
comparisons. But for uncertainty quantification and for method improvement, the opposite is more 
important. 

• Reisenthel (1) - Error has a quasi-physical form. This a cautionary statement since it means that the 
very areas of greatest interest in the response domain (boundaries, high gradients, near singularities, 
and so on) are likely to be the most in error. 

• Raj (PI) - Extensive [validation] correlations on geometries and flow conditions that differ 
substantially from those being considered by the design teams are of little value. This seems to be the 
experience of anyone who uses computational simulation in a development project. It suggests that 
any such project must carefully consider a progression of validation experiments during the project 
development. See Paper 26. 

• Eussen (4) - [We] need a very > experienced person ... when doing non-linear simulation. Despite the 
best efforts of method developers and the fond hopes of project managers, experience is still 
invaluable in not only getting good answers but in knowing how good they are. 

• Stem (21) - We’re too far from the asymptotic range. This lament was made by many others and 
leads the authors to wonder if our hopes in using Richardson extrapolation for numerical error 
estimation are misplaced. 

The authors also felt that several important questions were still unanswered: 

• How do we know that we have discretized sufficiently to capture the necessary physics for a complex 
application in a predictive high-consequence mode so that uncertainty has any reasonable meaning? 

• How do we deal with highly-correlated (distributed) errors? Not all of the possible realizations 
between those nice smooth uncertainty bounds are physically possible. 

• In the future (see Tinoco quote), all data bases for design will be built by simulation by necessity. 
How does that change the use of experimental facilities that were built and are being used now to 
build those data bases in the old (experimental) way? Will they be used strictly for validation 
purposes? If so, how would they be used to qualify predictions? 

• The future for much of design is unsteady simulation, even for CFD. It will dictate a different future 
for uncertainty. Will we have to use reduced-order models to be able to even talk about uncertainty? 

• Are we going to have to develop methods for characterizing the experimental uncertainties for 
parameter estimation of material properties that are rate dependent? 
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